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1.0  SUMMARY 


A  key  challenge  to  integrating  UAS  into  the  National  Airspace  System  (NAS)  is  providing  a 
reliable  means  for  UAS  to  safely  avoid  proximal  aircraft  through  the  development  of  a 
specialized  system  that  would  reside  onboard  the  UAS.  Although  this  system  has  to  be  capable 
of  operating  fully  autonomously,  in  most  cases  it  is  expected  that  the  operator  will  have  the 
opportunity  to  engage  with  the  system  regarding  situation  assessment  and  response  tasks  via  a 
SAA  display.  Given  the  increasing  sophistication  of  algorithms  underlying  this  new  capability, 
experimental  evaluation  of  display  transparency  and  automation  initiation  is  necessary  in  order  to 
establish  requirements  for  adequate  human-automation  system  performance.  The  study  described 
herein  was  an  operator-in- the  loop  assessment  supporting  the  development  of  a  Sense  and 
Avoid  (SAA|  display  that  enables  effective  teaming  of  an  Unmanned  Aerial  Systems  (UAS| 
operator  with  an  advanced  SAA  maneuver  algorithm  to  safely  avoid  proximal  air  traffic.  This 
study  examined  performance  differences  between  candidate  SAA  display  configurations  while 
UAS  operators  controlled  a  UAS  through  airspace  including  several  proximal  aircraft.  Twenty- 
two  participants  compared  three  stand-alone  SAA  displays  (Banding.  Probing  &  Dual 
Perspective)  across  two  automation  thresholds  (Well  Clear  &  Near  Mid-Air  Collision  (NMAC)). 
The  advanced  SAA  algorithm  maneuvered  significantly  later  than  the  pilots  but  it  did  so  in  a 
similar  fashion.  The  Well  Clear  automation  threshold  resulted  in  significantly  less  time  spent  in 
violation  of  traffic  when  compared  to  the  NMAC  automation  threshold.  Although  algorithm 
transparency  features  implemented  on  novel  displays  did  not  result  in  significant  performance 
differences,  participant  feedback  provided  valuable  insight  into  the  potential  utility  and 
drawbacks  of  each.  Further  research  w  ill  aim  at  increasing  the  complexity  of  engagements  and 
operational  scenarios  where  increased  algorithm  transparency  may  be  of  use  and  help  tease  apart 
any  existing  differences  amongst  SAA  displays. 

2.0  INTRODUCTION 

Current  DoD  Unmanned  Aerial  Systems  (UAS)  have  missions  in  every  type  of  airspace,  whether 
for  training,  transiting  operations  within  the  US.  or  for  operational  or  humanitarian  missions 
overseas.  Thus,  the  need  for  UAS  to  access  manned  airspace  already  exists  and  the  requirement 
will  only  grow  as  the  number  of  UAS  and  their  associated  missions  expand.  A  major 
technological  barrier  preventing  UAS  from  freely  operating  in  the  US  National  Airspace  (NAS) 
is  the  lack  of  a  "Sense-and-Avoid"  (SAA)  capability,  i.e..  the  ability  to  detect  and  safely  avoid 
other  aircraft  in  flight  (Cook  &  Davis.  2013).  In  order  to  increase  UAS  flight  safety  and  support 
UAS  integration  into  the  NAS  and  international  airspace,  several  platform  agnostic  SAA 
capabilities  consisting  of  sensors,  specialized  algorithms,  and  displays  have  been  developed. 

A  bulk  of  the  SAA  display  capabilities  that  have  been  established  utilize  simplistic  SAA 
algorithms  providing  intruder  threat  level  and  single  step,  one  dimensional  suggestive  maneuver 
guidance  (i.e..  turn  to  heading  095  or  climb  1000  ft.).  Several  research  efforts  have  attempted  to 
identify  the  necessary  display  information  needed  to  facilitate  adequate  avoidance  and  airspace 
separation  with  these  single-dimensional  systems.  In  one  such  study.  Friedman-Berg.  Rein  & 
Racine  (2014)  compared  four  display  categories  differing  in  information  level  and  concluded  that 
the  information  present  in  the  "prediction"  display,  consisting  of  intruder  color  coding  and  30 
second  dead  reckoning  vector  lines,  was  the  minimum  visual  information  required  for  a  SAA 
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pilot  display.  Intruder  color  coding  and  dead  reckoning  lines  resulted  in  significantly  less  Near 
Mid-Air  Collisions  (NMACs)  than  the  other  lower  level  displays.  Bell  (2012)  compared  a  basic 
SAA  display  to  two  "advanced"  concepts  depicting  the  relative  Closest  Point  of  Approach  (CPA) 
and  ownship  avoidance  areas  via  polygon  shapes.  Although  the  frequency  of  Well  Clear 
violations  did  not  significantly  differ  between  the  three  display  concepts,  the  basic  display 
resulted  in  significantly  longer  violations  than  the  two  advanced  displays.  Similarly,  the  National 
Aeronautics  and  Space  Administration  (NASA)  investigated  the  effect  of  information  level  and 
display  location  on  UAS  pilots'  performance  on  maintaining  self-separation  and  collision 
avoidance  from  other  aircraft  while  operating  in  civil  airspace  (Fern.  Rorie.  Pack.  Shively  & 
Draper.  2015).  Response  time  analyses  revealed  significantly  faster  total  response  times  with  the 
advanced  information  displays,  which  included  additional  situational  awareness  information  and 
maneuver  guidance,  than  the  basic  information  displays.  These  types  of  studies  have  allowed  for 
a  minimum  information  set  to  be  established  and  implemented  into  several  initial  SAA 
capabilities.  All  these  studies  provide  evidence  that  SAA  algorithmic  transparency,  which  is 
characterized  by  observability,  directability.  adaptability,  and  broadening  (DePass  et  a!.,  2011; 
Truxler  et  al..  2012).  is  an  important  factor  for  successful  employment  of  automation. 

While  these  SAA  display  capabilities  have  proven  to  be  effective  when  utilizing  relatively 
simplistic  SAA  algorithms,  increasingly  complex  SAA  algorithms  are  being  created  to  provide 
pilots  with  additional  maneuver  information  and  autonomous  control  capabilities.  Consequently, 
the  previously  established  minimum  information  set  needs  to  be  reinvestigated  for  those  displays 
that  will  utilize  these  more  advanced  algorithms  to  ensure  adequate  transparency  and  thus  proper 
automation  use  (Parasuraman  &  Riley,  1997).  Bihrle’s  Jointly  Optimal  Conflict  Avoidance 
(JOCA)  algorithm  is  one  such  algorithm  (Graham  et  al.,  20 1 1  >.  Using  a  unique  “2 -stitch"  probing 
mechanism.  JOCA  provides  sophisticated  suggestive  maneuver  guidance  consisting  of  tw  o  step, 
multidimensional  maneuvers  (i.e..  turn  to  heading  075  and  then  descend  500  ft.).  If  the 
engagement  becomes  imminent  and  the  pilot  does  not  take  appropriate  action.  JOCA  executes  an 
autonomous  avoidance  maneuver  to  resolve  the  conflict.  These  additional  capabilities  offered  by 
JOCA  lend  themselves  to  novel  display  concepts  that  convey  additional  transparency,  therefore, 
in  an  effort  to  explore  the  utility  of  this  additional  transparency,  the  follow  ing  study  designed  and 
evaluated  two  novel  SAA  suggestive  displays  against  a  proven  SAA  display  design  previously 
developed  for  more  simplistic  SAA  algorithms  (Pack,  Draper.  Darrah.  Squire  &  Cooks.  2015). 

While  transparency  is  an  imperative  factor  for  human  trust  in  automation  other  factors  such  as 
behavior  familiarity  are  also  important  contributing  factors  (Chen  et  al..  2014;  Lee  &  See.  2004). 
Autonomous  (light  systems  that  align  with  pilot  training  and  behavior  tend  to  support  stronger 
trust  perceptions  from  pilots  (Lyons  et  al..  2016).  With  this  in  mind,  this  study  compared  the 
participants'  maneuver  behavior  to  the  SAA  algorithm’s  maneuver  behavior.  Lastly,  recognizing 
that  the  utility  of  a  SAA  display  might  be  effected  by  temporal  implementation,  two  different 
thresholds  were  implemented  for  when  automation  took  control  for  a  response  recovery. 

The  objective  of  the  current  study  was  to  examine  the  display  and  performance  implications 
associated  with  the  use  of  advanced  SAA  algorithms  such  as  JOCA.  Several  hypotheses  were 
formulated  based  on  past  research:  ( 1 )  those  SAA  displays  designed  to  provide  advanced 
algorithm  transparency  (Dual  Perspective  &  Probing)  would  outperform  the  display  designed 
with  transparency  associated  with  a  more  simplistic  algorithm.  (2)  initiating  automation  support 
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early  (Well  Clear  Threshold)  versus  late  (NMAC  Threshold)  will  result  in  increased  avoidance 
performance,  and  (3)  given  the  SAA  algorithm  was  designed  for  purely  autonomous 
implementation  (no  human  oversight),  its  solution  recommendations  will  be  dissimilar  to  that  of 
a  human  operator’s  response  to  similar  traflic  engagements. 

3.0  METHOD 

3.1.  Participants 

Twenty-two  male  pilots  (eight  RQ-4.  eight  MQ-l/MQ-9.  and  six  manned  aircraft;  mean  age  38  ± 
9  years)  were  recruited  for  this  experiment.  Participants  had  an  average  of  2.616  overall  flight 
hours  and  1.287  Unmanned  Aircraft  System  (UAS)  hours.  Participation  was  limited  to  military 
rated  pilots  and  civilian  pilots  w  ith  full  Instrument  Flight  Rules  (IFRi  ratings.  Recent  experience 
(within  the  last  two  years)  flying  within  controlled  airspace  was  required  as  well.  Each 
participant  read  and  signed  an  informed  consent  form  approved  by  the  Wright  Patterson  Air 
Force  Base  Institutional  Review  Board  (IRB). 

3.2.  Experimental  Design 

Participants  were  tasked  with  operating  a  simulated  UAS,  "I  I A  WK2 1 along  a  flight  path  with  a 
set  mission  altitude  of  12.000  feet.  Furthermore,  participants  were  responsible  for  maintaining 
safe  separation  as  if  they  were  operating  in  the  NAS  and  responding  to  a  variety  of  scripted 
health  and  status  tasks.  A  3  x  2  within-subjects.  repeated  measures  factorial  design  was  used  to 
compare  the  effect  of  display  type  and  automation  threshold  on  UAS  pilots'  performance  on 
maintaining  Well  Clear  from  other  aircraft  while  operating  in  the  NAS.  Three  suggestive  SAA 
Displays  (Banding.  Probing.  &  Dual  Perspective)  were  compared  across  two  Automation 
Thresholds  (Well  Clear.  NMAC).  thus  participants  completed  a  total  of  six  trials.  These  display 
conditions  are  described  below. 

3.3  Simulation  Environment 

3.3.1.  (.round  Control  Station.  Participants  were  situated  at  the  Air  Force  Research 
Laboratory's  Vigilant  Spirit  Control  Station  (VSCS)  (Figure  1 ).  VSCS  is  a  mature  GCS  operator 
interface  designed  to  support  the  control  of  UAS  and  their  associated  payloads  (Pat/ek  et  al„ 
2009).  In  this  study.  VSCS  generated  three  separate  pilot  displays  in  the  ground  station:  1 ) 
Tactical  Situation  Display  (TSD).  2)  health  and  status  panel,  and  3)  SAA  traflic  display.  The  TSD 
served  as  the  pilot's  primary  control  interface,  providing  ownship  and  route  information,  a 
moving  map,  and  ownship  control  capabilities.  The  health  and  status  panel  contained  telemetry 
data,  a  chat  room,  a  backup  radio  communication  panel,  an  electronic  checklist,  and  a  subsystem 
annunciator  panel.  The  SAA  display,  which  varied  by  trial,  was  populated  with  traflic 
information  for  those  aircralt  in  relative  proximity  to  the  ownship. 
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Figure  1.  Vigilant  Spirit  Control  Station 

3.3.2  Traffic  Simulation.  The  Air  Force  Research  Laboratory's  Vigilant  Spirit  (  VS)  simulation 
capability  was  used  to  generate  the  air  traffic  environment  tor  this  study.  The  VS  simulation 
capability  allowed  for  six  unique  traffic  encounters  to  be  generated  and  repeated  from  participant 
to  participant.  A  mix  of  geometries  including  head-on.  side,  and  overtake  encounters  were 
incorporated  into  the  traffic  encounters. 

3.3.3  Sense  and  Avoid  Algorithm.  For  this  study,  the  Air  Force  utilized  a  SAA  Maneuver 
Algorithm  developed  by  Bihrle  known  as  the  Jointly  Optimal  Conflict  Avoidance  (JOCA) 
algorithm  (Graham  et  al..  2011).  In  addition  to  evaluating  and  categorizing  surrounding  traffic 
according  to  their  predicted  threat  level  relative  to  ownship.  the  SAA  maneuver  algorithm  is 
capable  of  providing  suggestive  maneuver  guidance  and  executing  autonomous  avoidance 
maneuvers.  The  viability  of  potential  avoidance  maneuver  resolutions  are  tested  using  a  ’*2- 
stitch"  probing  mechanism  in  which  an  initial  series  of  maneuv  er  resolution  probes  spawn  a 
secondary  series  of  maneuver  resolution  probes  (Figure  2). 

I M  Pnltr  Serif*  2nd  Prollf  Sritrs 


Figure  2.  SAA  Maneuver  Algorithm  ••2-Stltch”  Probing  Mechanism 
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The  resulting  maneuver  resolution  probes  are  categorized  according  to  a  hierarchy  of  constraints 
to  produce  a  "branch- like”  network  of  potential  maneuver  options.  Maneuver  options  include  not 
only  heading  and  altitude  maneuvers  but  also  two-step  maneuvers  (e.g.  turn  to  heading  030  and 
then  climb  500  ft.)  and  combination  maneuvers  (e.g.  turn  to  heading  050  while  descending  1.000 
ft.).  For  this  study,  the  SAA  maneuver  algorithm  detected  and  color  coded  traffic  based  on  a  set 
alerting  structure  (Table  1 ).  Maneuver  probes  were  utilized  to  initially  provide  potential 
maneuver  resolutions  for  the  pilots  to  choose  from.  If  the  situation  progressed  to  a  certain 
threshold  where  the  algorithm  had  to  take  over,  maneuver  probes  were  used  to  determine  and 
execute  an  autonomous  avoidance  maneuver  that  maintained  safety  of  flight  while  also 
minimizing  deviation  from  mission  path.  The  automation  threshold,  dictating  when  the  SAA 
maneuver  algorithm  would  take  control  of  the  aircraft,  was  systematically  varied  in  the  study  (as 
described  below).  With  an  update  rate  of  1  HZ.  the  SAA  displays  were  populated  and  then 
continually  updated  to  reflect  the  latest  data  output  from  the  SAA  maneuver  algorithm. 


Alert/Tlireat 

Level 

Horizontal 
Threshold  (nm) 

Vertical 

Threshold 

(feet) 

Time 

Threshold 

Color 

Proximate 

2.0  nm 

1,000  ft 

N/A 

White 

Preventative 

.66  nm 

>450  to  <700  ft 

<  120  sec 

White  with  circle 

Well  Clear 

.66  nm 

450  ft 

<  110  sec 

Yellow/Yellow 
w/red  outline 

NMAC 

.08  nm 

100  ft 

<  40  sec 

Red 

Table  1.  Multi-Level  Alerting  Structure 


3.4  Stimuli 

3.4.1  Automation  Thresholds.  Automation  thresholds  were  systematically  varied  to  either 
allow  a  larger  degree  of  separation  between  aircraft  to  be  maintained  (Well  Clear)  or  a  smaller 
degree  of  separation  (Near  Mid-Air  Collision  (NMAC)).  When  the  automation  threshold  was  set 
to  Well  Clear,  the  SAA  maneuver  algorithm  would  take  over  to  avoid  a  "Well  Clear  violation”, 
which  was  defined  as  having  a  predicted  horizontal  miss  distance  (HMD)  less  than  or  equal  to 
4000  feet  (0.66  nautical  miles),  a  current  vertical  separation  less  than  or  equal  to  450  feet,  and  a 
modified  tau  (a  metric  of  time  to  closest  approach  in  the  horizontal  plane)  less  than  or  equal  to  35 
seconds.  When  the  automation  threshold  was  set  to  “NMAC”,  the  pilot  was  solely  responsible 
for  maintaining  Well  Clear  separation  but  the  SAA  maneuver  algorithm  would  take  over  to  avoid 
a  NMAC  violation,  which  was  defined  as  having  a  current  HMD  less  than  or  equal  to  .08 
nautical  miles  and  a  current  vertical  separation  less  than  or  equal  to  100  feet. 

3.4.2  SAA  Displays.  Three  SAA  displays  were  designed  and  tested  in  this  study.  The  Banding 
Display  proved  to  be  an  effective  display  in  previous  studies  (Pack.  Draper.  Darrah,  Squire  & 
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Cooks,  2015)  (Figure  3).  In  addition  to  intruder  alerting,  a  color  coded  arc  and  altitude  tape  were 
included  to  continually  show  ranges  of  potential  heading  and  altitude  resolutions,  respectively. 
The  arc  on  the  inner  range  ring  represented  a  180°  range  of  potential  heading  maneuver 
resolutions  and  the  altitude  tape,  consisting  of  five  blocks,  represented  potential  elevation 
resolutions  in  500  foot  increments  +/-  1000  feet  from  the  ounship  s  current  flight  level.  Color 
indicated  the  predicted  quality  or  safety  of  a  heading  or  altitude  maneuver  resolution  (green  no 
conflicts,  yellow  predicted  Well  Clear  violation,  red  predicted  NMAC  violation).  The  arc  and 
altitude  tape  reflected  the  SAA  maneuver  algorithm's  la  series  of  probes.  Secondary  probes  were 
not  factored  into  this  display 


Figure  3.  Banding  Display 

The  Probing  Display  was  a  design  concept  that  emphasized  the  SAA  maneuver  algorithm's  "2- 
stitch"  probing  logic  (Figure  4).  In  addition  to  intruder  alerting  (same  as  used  in  Banding 
Display),  a  color-coded  series  of  dots  and  a  vertical  stack  were  included  to  show  potential 
discrete  heading  and  altitude  maneuver  resolutions,  respectively.  The  nine  dots  on  the  inner  range 
ring  represented  ownship  potential  headings,  with  four  potential  heading  resolutions  to  the  right 
and  four  potential  heading  resolutions  to  the  left  of  a  current  heading  resolution.  The  vertical 
stack,  consisting  of  four  arrows  and  the  current  heading  resolution  dot.  represented  potential 
elevation  resolutions  in  500  foot  increments  */-  1000  feet  from  the  ownship 's  current  flight  level. 
Color  once  again  indicated  the  predicted  quality  or  safety  of  a  heading  or  altitude  maneuv  er 
resolution.  The  dots  and  vertical  stack  on  the  inner  range  ring  reflected  the  SAA  maneuver 
algorithm's  1st  series  of  probes.  Insight  into  the  SAA  maneuver  algorithm's  2nd  series  of  probes 
was  provided  via  a  “pimento”  display  feature  and  an  interactive  probing  capability.  The 
“pimento”  display  feature  distinguished  a  heading  or  altitude  resolution  with  a  yellow  center  if  it 
was  predicted  to  result  in  a  Well  Clear  v  iolation  based  on  the  2nd  series  of  probes.  Triggered  via 
a  button  click,  the  interactive  probing  capability  expanded  upon  the  "pimento"  display  feature  by 
prov  iding  a  second  series  of  dots  and  vertical  stack  for  a  specific  heading  or  altitude  maneuver 
resolution.  This  second  set  of  dots  and  arrows  reflected  one  of  the  SAA  maneuver  algorithm's 
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2nd  probes,  allowing  operators  to  gain  detailed  insight  into  die  entire  series  of  probes  for  a 
particular  heading  or  altitude  resolution.  By  allowing  die  operator  to  observe  and  direct  more  of 
the  algorithm  functionality  associated  widi  the  two  probes,  algorithm  transparency  was  diought 
to  be  increased,  dius  we  expected  to  see  an  increase  in  performance. 


Figure  4.  Probing  Display 

The  Dual  Perspective  Display  was  a  novel  design  concept  that  emphasized  the  SAA  maneuver 
algoridim's  capability  to  provide  combination  maneuver  resolutions  (Figure  5).  In  addition  to 
intruder  alerting,  a  color  coded  series  of  squares  w  ithin  a  cockpit  view1  was  included  to  show 
potential  heading,  altitude,  and  combination  maneuver  resolutions.  The  nine  squares  on  the  x- 
axis  represented  the  ownship’s  current  heading  (center),  four  potential  heading  resolutions  to  the 
right,  and  four  potential  heading  resolutions  to  the  left.  The  five  squares  on  the  y-axis  represented 
potential  elevation  resolutions  in  500  fool  increments  +/-  1000  feet  from  the  ownslup's  current 
flight  level.  Combination  maneuver  resolutions  were  represented  by  those  squares  that  didn't  full 
on  a  specific  axis  but  rather  within  a  quadrant.  Color  once  again  indicated  the  predicted  quality 
or  safety  of  a  heading  or  altitude  maneuver  resolution.  The  squares  reflected  the  SAA  maneuver 
algorithm's  1st  series  of  probes.  Similar  to  the  Probing  Display,  insight  into  the  SAA  maneuver 
algoridim's  2nd  series  of  probes  was  pro\  ided  via  the  "pimento”  display  feature.  Once  again, 
bringing  out  more  features  and  functionalities  of  the  algorithm,  particularly  combo  maneuvers, 
for  the  pilot  to  work  with  was  thought  to  bolster  transparency. 


7 

DISTRIBUTION  STATEMENT  A.  Approved  (or  public  release. 


Cleared.  88PA.  Case  #2017-2201 . 


Figure  5.  Dual  Perspective  Display 


3.4.3  Transition  to  Automation  Display  Features.  Display  features  accompanying  the 
transition  from  traditional  flight  control  to  autonomous  control  were  developed  and  remained 
constant  across  each  display  type  as  a  means  of  informing  the  pilot  about  algorithm  intentions 
and  operations  (Figure  6).  Throughout  the  transition  pilots  were  provided  with  ( I  ( a  countdown 
to  automation  timer.  (2)  an  avoidance  notification  accompanied  by  an  auditory  alert.  (3)  a 
returning  to  course  notification,  and  |4|  an  indication  of  maneuver  completion.  In  addition,  a 
vector  was  provided  off  the  nose  of  the  ownship  to  indicate  the  SAA  algorithm's  intent. 


(1) 

(2) 

(3) 

(4) 


rancitioninq  to  Automation  in 


tl I 


Avoiding 
I  Wl 

Maintain  120M  ft 


Returning  to  Course 
Turn  to  i  ufca 

Maintain  U'ftTO  ft 


Maneuver  Complete 
MMntan  T  IMM 
Maintain  121CO  ft 


Figure  6.  Transition  to  Automation  Display  Features 
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3.5  Procedure 


Upon  entry  into  the  lab.  participants  read  and  signed  an  informed  consent  form  and  completed  a 
demographics  survey,  which  elicited  information  regarding  their  flight  experience.  Participants 
were  provided  an  introduction  to  the  study  and  their  operator  responsibilities.  Responsibilities 
included  flying  “1 IAWK2 1  ”  along  a  flight  plan,  responding  to  traffic  alerts  when  necessary,  and 
performing  health  and  status  tasks  as  they  appeared.  Initially,  training  was  provided  on  the  basic 
functionality  of  VSCS,  including  how  to  use  the  TSD  to  perforin  heading  and  altitude  holds. 
Participants  were  not  required  to  gain  simulated  Air  Traffic  Control  (ATC)  clearance  prior  to 
maneuvering  during  this  study  but  were  encouraged  to  return  to  the  flight  plan  as  soon  as 
practical  after  maneuvering  for  traffic.  After  being  taught  the  definitions  of  Well  Clear  and 
NMAC,  the  participants  received  training  on  the  multi-level  alerting  structure.  The  next  portion 
of  training  focused  on  the  SAA  maneuver  algorithm.  Lastly,  participants  were  trained  on  each 
suggestive  display  and  automation  threshold.  To  account  for  training  and  fatigue  effects, 
automation  threshold  and  suggestive  SAA  display  order  were  assigned  based  on  a 
counterbalance. 

Participants  saw  the  three  suggestive  SAA  displays  across  both  automation  thresholds  for  a  total 
of  six  6-minute  trials.  Six  unique  traffic  engagements  involving  either  one  or  two  intruders  were 
randomized  in  such  a  way  that  each  suggestive  SAA  display  was  paired  with  each  traffic 
engagement  an  equal  number  of  times.  Following  the  completion  of  each  trial,  participants 
completed  a  subjective  questionnaire,  which  garnered  feedback  on  the  usefulness  of  each  display 
and  the  autonomous  avoidance  maneuver  as  well  as  display  ranking  data.  After  completing  all 
six  trials,  participants  completed  a  final  questionnaire  allowing  participants  to  score  individual 
aspects  of  each  display. 

3.6  Data  Analysis 

Data  was  analyzed  utilizing  multiple  repeated-measures  Analyses  of  Variance  (ANOVAs),  t-tests. 
and  chi-square  goodness-of-fit  tests.  The  dependent  variables  included  Well  Clear  violation  rate, 
time  from  maneuver  input  to  predicted  loss  of  Well  Clear  (LoWC).  and  subjective  display 
rankings.  An  alpha  level  of  .05  was  used  for  all  analyses  to  establish  statistical  significance. 

4.0  RESULTS 

4.1  Objective  Results 

4.1.1  Well  Clear  Violations.  Well  Clear  violation  rate  was  used  as  the  primary  measure  of 
performance  in  this  study.  A  3  (display  type:  Banding.  Probing  &  Dual  Perspective)  X  2 
(automation  threshold:  Well  Clear  &  NMAC)  repeated-measures  ANOVA  revealed  that  neither 
display  type,  F(2.  42)  .42.  p  >  .05,  nor  automation  threshold.  F(  1,  21)  .  1 9.  p  >  .05.  had  a 
significant  effect  on  performance  (Figure  7).  Overall.  Well  Clear  violations  were  rare  with  only 
13.6%  of  trials  resulting  in  a  violation  of  Well  Clear. 
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Figure  7.  Prevalence  of  Well  Clear  Violations 


That  being  said,  a  2  sample  t-test  revealed  that  of  those  Well  Clear  violations  that  occurred  more 
time  was  spent  in  violation  of  Well  Clear  when  the  automation  threshold  was  set  at  NMAC  rather 
than  Well  Clear,  t  4.73  df  7,  p  <  .05  (Figure  8}.  Thus,  earlier  onset  of  automation  support 
enabled  less  time  in  violation  of  Well  Clear  space. 


Figure  8.  Time  Spent  in  Violation  of  Well  Clear 

4.1.2  Pilots  vs.  SAA  Algorithm  Maneuver  Behavior.  Although  the  participants  tended  to 
maneuver  earlier  than  the  automation,  the  type  of  maneuvers  being  executed  by  the  participants 
and  the  automation  were  similar.  When  further  investigated  the  automation  was  found  to 
maneuver  in  the  same  direction  as  the  participants  in  five  out  of  the  six  traffic  engagements. 
Figure  9  shows  the  participants  maneuver  choices  in  comparison  to  JOCA's  maneuver  choice  for 
Engagement  2.  which  involved  two  intruders  approaching  from  the  same  side  of  ownship.  The 
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participant's  maneuver  choices  are  represented  by  dots  and  JOCA's  maneuver  choice  is 
represented  by  a  triangle.  Of  the  20  participants  who  responded  to  this  engagement,  ten  chose  to 
climb  500  feet  to  avoid  the  traffic.  JOCA  very  similarly  chose  to  slightly  turn  to  the  right  and 
climb  500  feet. 


Figure  9.  Pilot  and  SAA  Maneuver  Algorithm  Maneuver  Choice  Breakdown  for 

Engagement  2 


4.2  Subjective  Results 

4.2.1  Suggestive  Display  Rankings.  After  completing  the  study  participants  ranked  the 
suggestive  displays  from  1-3  in  terms  of  preference  with  1  being  their  least  favorite  and  3  being 
their  favorite.  A  one-way  ANOVA  revealed  display  type  did  not  have  a  significant  effect  on 
display  ranking.  F(2.  63)  2.24.  p  >  .05  (Figure  10). 


Figure  10.  Average  Display  Rankings 

4.2.2  Automation  Threshold  Preference.  Upon  completion  of  the  study  participants  were  also 
asked  whether  they  preferred  the  Well  Clear  or  NMAC  automation  threshold.  Although  the 
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preference  was  not  significant,  of  the  20  participants  who  responded  eight  preferred  the  Well 
Clear  threshold  and  1 2  preferred  the  NM AC  threshold,  t  .89  df  1 9.  p  >  .05. 


4.2.3  Transition  to  Automation  Display  Features.  Participants  scored  the  transition  to 
automation  display  features  from  1-4  in  terms  of  usefulness  with  1  being  “Not  Useful"  and  4 
being  “Extremely  Useful”.  A  chi-square  goodness-of-fit  w  as  performed  to  determine  that  scores 
were  not  equally  distributed  in  the  population.  X2(3.  N  20)  10.8.  p  <  .05  with  the  majority  of 

participants  scoring  the  transition  to  automation  display  elements  as  "Extremely  Useful”  (Figure 

ID- 


Scoro  the  transition  to  automation  information  elements  in 
terms  of  usefulness 


Not  Useful  Eitrcmcly  Useful 

Figure  II.  Transition  to  Automation  Display  Feature  Scores 

When  asked  if  any  additional  information  should  be  provided  on  the  DAA  display  to  facilitate  a 
successful  transfer  of  control,  participants  said  they  would  like  to  know  not  only  when  the  SAA 
maneuver  algorithm  is  going  to  take  over  but  also  what  maneuver  it  plans  to  execute.  With  that 
additional  piece  of  information,  the  participants  requested  the  ability  to  execute  the  planned 
maneuver  prior  to  the  countdown  reaching  zero  via  a  button  click.  Lastly,  they  said  they  would 
like  the  aural  cues  to  complement  the  textbox  and  vectors  by  indicating  the  SAA  maneuver 
algorithm's  commanded  maneuver  such  as  "climbing,  climbing". 

4.2.4  Automated  Response  Ratings.  Participants  were  additionally  tasked  with  providing 
subjective  feedback  on  the  effectiveness  of  the  automated  maneuver.  Figure  12  shows  the 
participants'  responses  when  asked  to  score  their  agreeableness  with  the  statement  “The 
automated  maneuver  was  able  to  maintain  adequate  separation".  A  chi-square  goodness-of-fit 
was  performed  to  determine  that  scores  were  not  equally  distributed  in  the  population.  X2<4.  N  = 
22)  =  14.36.  p  <  .05  (Figure  14)  with  the  majority  of  participants  indicating  that  they  “Somewhat 
Agreed"  with  the  statement.  Written  explanations  such  as  “The  auto  maneuver  made  good 
decisions  about  how  to  maneuver,  just  not  timely”  indicate  that  participants  could  agree  with  the 
SAA  algorithm's  chosen  maneuver  but  felt  the  maneuver  wasn't  executed  early  enough. 
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The  automated  maneuver  was  able  to  maintain  adequate 
separation. 
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Agree 

Figure  12.  Automated  Response  Ratings 


5.0  DISCUSSION 

5.1  Hypothesis  1 :  Display  Transparency 

Our  first  hypothesis  suggesting  that  those  displays  with  enhanced  algorithm  transparency  would 
outperform  the  display  with  less  insight  into  the  advanced  SAA  algorithm  was  not  supported. 
Objectively,  all  three  suggestive  displays  performed  the  same  despite  varying  look  ahead  times 
and  combo  maneuver  depiction  capabilities.  The  probing  and  dual  perspective  displays  tended  to 
be  ranked  higher  by  participants  than  the  banding  display  but  this  preference  was  not  significant. 
The  overall  low  level  of  well  clear  violations  implies  that  each  display,  even  the  more  basic 
banding  display,  afforded  the  participants  with  a  proper  level  of  understanding  in  regards  to  the 
maneuver  algorithm's  intent,  performance,  future  plans,  and  reasoning  process.  The  banding 
display  might  have  been  more  basic  and  one-dimensional  in  terms  of  maneuver  guidance 
presentation  but  the  transparency  "threshold”  needed  to  result  in  appropriate  automation  usage, 
which  is  defined  as  both  high  rates  of  proper  use  when  automation  is  accurate  and  high  rates  of 
proper  disuse  when  it's  not.  was  met  (Chen  et  al.,  2014).  This  in  turn  mitigated  the  decrease  in 
performance  we  expected  to  see.  The  data  suggests  that  those  displays  designed  and  successfully 
tested  with  less  advanced  SAA  algorithms  in  mind  still  have  merit  even  with  more  advanced 
SAA  algorithms.  That  being  said,  it's  also  possible  that  the  transition  to  automation  features, 
which  were  included  in  each  display,  provided  adequate  transparency  information  and  display 
effects  were  washed  out  as  a  result.  Future  studies  plan  to  examine  these  displays  under  more 
stressful  conditions  that  could  reveal  display  shortcomings.  Such  conditions  might  include  a 
greater  number  of  intruders  per  encounter  with  dynamically  varying  routes,  military  airspace 
environments,  uncertainty  variables,  and  increased  operator  workload. 
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5.2  Hypothesis  2:  Automation  Thresholds 


In  regard  to  when  the  SAA  algorithm  would  step  in  and  autonomously  maneuver  the  aircraft,  we 
hypothesized  that  automation  support  at  the  farther  out  Well  Clear  threshold  would  lead  to  belter 
avoidance  performance  than  if  this  were  to  occur  at  the  closer  in  NMAC  threshold.  Though  well 
clear  violation  prevalence  was  the  same  across  both  automation  thresholds,  this  hypothesis  was 
supported  by  the  time  spent  in  violation  of  Well  Clear  data.  When  those  Well  Clear  violations 
that  occurred  were  further  investigated,  time  spent  in  violation  of  Well  Clear  was  found  to  be 
significantly  less  when  the  SAA  algorithm  was  available  to  assist  participants  at  the  Well  Clear 
automation  threshold  instead  of  the  NMAC  threshold.  This  finding  suggests  that  while  Well 
Clear  violations  might  be  rare,  when  they  occur  having  automation  available  at  the  Well  Clear 
threshold  to  assist  the  pilot  mitigates  the  severity  of  the  violation.  Future  research  may  examine 
adjustable  thresholds  of  automation  that  would  allow  operators  to  adjust  when  automation  would 
take  over  based  on  mission  constraints. 

5.3  Hypothesis  3:  Pilot  vs.  SAA  Algorithm  Behavior 

Lastly,  we  hypothesized  that  because  the  SAA  algorithm  used  in  this  study  was  originally 
designed  for  machine  to  machine  interaction,  the  participants  and  the  SAA  algorithm  would  react 
dissimilarly  to  traffic  engagements  in  terms  of  maneuver  time  and  maneuver  type.  The 
participants  did  indeed  maneuver  significantly  earlier  than  the  SAA  algorithm  in  relation  to  the 
predicted  loss  of  Well  Clear.  That  being  said,  when  the  type  of  maneuvers  being  performed  by 
the  SAA  algorithm  were  compared  to  the  participants'  maneuvers  across  each  engagement  they 
were  found  to  be  quite  similar.  This  piece  of  data  does  not  support  our  hypothesis.  Instead,  it 
suggests  that  although  the  SAA  algorithm  might  maneuver  later  for  enhanced  efficiency,  when  it 
does  actually  choose  to  maneuver  the  type  of  maneuvers  being  executed  are  similar  in  direction 
and  magnitude  to  operators'  maneuvers.  Feedback  on  the  transition  to  automation  features 
revealed  a  desire  for  a  display  function  that  would  allow  the  operator  to  execute  the  algorithm 
suggested  maneuver  prior  to  the  end  of  the  countdow  n,  imply  ing  once  again  that  pilots  tended  to 
agree  with  the  maneuver  type  but  not  the  timing  of  it.  Continued  support  and  research  of  the 
SAA  algorithm  will  focus  on  defining  the  operational  role  and  optimizing  for  human-machine 
interaction. 

6.0  CONCLUSIONS  AND  RECOMMENDATIONS 

The  current  study  demonstrated  the  display  and  performance  implications  of  integrating 
an  advanced  SAA  algorithm.  Despite  the  use  of  an  SAA  algorithm  with  advanced  features,  the 
baseline  display.  Banding,  performed  just  as  well  as  the  two  new  novel  display  concepts.  Based 
on  these  results,  the  additional  algorithm  transparency  features  included  in  the  Dual  Perspective 
and  Probing  displays  such  as  the  pimento  logic,  combo  maneuvers  and  2nd  stitch  probe 
observability,  while  useful,  cannot  be  constituted  as  being  a  requirement  for  adequate 
performance.  In  regards  to  performance  implications,  though  violations  were  consistent 
throughout  both  automation  thresholds,  a  deeper  analysis  revealed  that  the  Well  Clear  threshold 
paired  with  the  algorithm  resulted  in  less  time  spent  in  violation  of  Well  Clear.  Lastly,  despite 
the  algorithm  maneuvering  far  later  than  when  the  operators  preferred,  it  maneuvered  in  a  similar 
fashion.  Further  research  will  aim  at  creating  more  complex  engagements  and  operational 
scenarios  where  the  increased  transparency  may  be  of  use  and  help  tease  apart  any  existing 
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differences  amongst  SAA  displays.  Furthermore,  incorporation  of  adjustable  automation 
thresholds  rather  than  one  or  the  other  may  prove  to  be  the  best  solution  in  these  more  complex 
environments. 
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8.0  LIST  OF  ACRONYMS 


L'AS  =  Unmanned  Aerial  Systems 

NAS  National  Airspace  System 

SAA  Sense  and  Avoid  Display 

NMAC  Near  Mid-Air  Collision 

DoD  Department  of  Defense 

CPA  Closest  Point  of  Approach 

NASA  National  Aeronautics  and  Space  Administration 

JOCA  Jointly  Optimal  Conflict  Avoidance 

IFR  Instrument  Flight  Rules 

IRB  Institutional  Review  Board 

VSCS  Vigilant  Spirit  Control  Station 

TSD  Tactical  Situation  Display 

HMD  Horizontal  Miss  Distance 

LoWC  Loss  of  Well  Clear 

AN  OVA  Anal  ysis  of  Variance 

ATC  Air  Traffic  Control 
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